fix(ingest): continuous datasources are not validate_only - #253
fix(ingest): continuous datasources are not validate_only#253eddietejeda wants to merge 1 commit into
Conversation
new-datasource unconditionally set validate_only=true ("discover schema, don't
load"), so --continuous produced a spec with BOTH continuous and validate_only.
That is the exact combination the worker now rejects (422): the scheduler
re-runs it every tick but the drain falls to full-replace, reloading the whole
bucket forever. It was the root cause of the otel_telemetry incident.
A continuous datasource is a persistent, self-loading source the scheduler keeps
synced — the opposite of a one-off schema preview. Set validate_only only when
NOT continuous. Without this, --continuous is unusable against a worker carrying
the guard (dlthubworker #193): every attempt 422s.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| // worker rejects the pair 422, and before it did, the datasource was | ||
| // re-run every tick but fell to the full-replace path — reloading the whole | ||
| // bucket forever. So continuous datasources are created ready to sync. | ||
| req.validate_only = !req.continuous; |
There was a problem hiding this comment.
nit: the post-create UX still assumes a validate-only add (not blocking).
With validate_only = false, the create job for a --continuous datasource now actually seeds data, but run_source (src/commands/ingest.rs:1335-1343) still polls with the verb "discovering schema" and render_datasource_added finishes with Import data with: hotdata ingest new-import --source {source} --all. For a continuous source that hint is wrong — it's self-loading and the scheduler keeps it synced. Also, the default --wait-timeout of 300s was sized for schema discovery; an initial full seed of a large bucket can blow past it and exit 2 (recoverable via the printed status --wait hint, but surprising).
Threading the continuous flag down to pick the verb ("seeding datasource") and swap the trailing hint would make the success output match what actually happened.
| }; | ||
| // Adding a datasource discovers the schema only — never loads data. | ||
| req.validate_only = true; | ||
| // Adding a datasource discovers the schema only — never loads data — EXCEPT |
There was a problem hiding this comment.
super nit: skills/hotdata/SKILL.md:250 still says new-datasource "Validates credentials + discovers the schema; loads NO data", and the --continuous bullet at line 259 only describes the scheduled re-runs. After this change the --continuous add does load data up front. Worth a clause on line 259 noting the create seeds the initial load (not blocking).
There was a problem hiding this comment.
Fix is correct and scoped: only the flag-driven path sets continuous (the wizard never does), so validate_only = !req.continuous leaves every other family and the wizard path unchanged. Test assertions cover both directions. Two non-blocking notes inline.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
1 participant
Urgent:
--continuouscurrently 422s in the fieldnew-datasourcesetreq.validate_only = trueunconditionally ("adding a datasource discovers the schema only"), so--continuousproduced a spec carrying bothcontinuousandvalidate_only.That is exactly the combination the worker now rejects with a 422 (dlthubworker #193, deployed): the scheduler re-runs such a datasource every tick but the drain falls to the full-replace path — reloading the whole bucket forever. It was the root cause of the
otel_telemetryincident (silently, before the guard).Now that #193 is live in prod,
hotdata ingest new-datasource --continuousfails 422 every time. This restores it.Fix
A continuous datasource is a persistent, self-loading source the scheduler keeps synced — the opposite of a one-off schema preview. So set
validate_onlyonly when not continuous:--continuous(incl. shaped--record-shape otel_traces) now creates a ready-to-sync datasource — verified end to end against prod: the re-registeredotel_telemetryseeded via the incremental path and landed 93k spans.create_request_filesystem_carries_continuous_flagstrengthened to assert!validate_onlyfor continuous andvalidate_onlyfor a plain add.Release
Should ship promptly as a patch (0.23.1) —
--continuousis broken in 0.22.0/0.23.0 against the current worker.